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METHOD AND APPARATUS FOR DATA TRANSFER AND STORAGE IN A 
HIGHLY PARALLEL COMPUTER NETWORK ENVIRONMENT 

5 

RELATED APPUCATION5 
This application is related to co-pending application entitled "METHOD AND 
APPARATUS FOR TRANSFERRING AND STORING DATA FROM AN 
ARBITRARILY LARGE NUMBER OF NETWORKED COMPUTER STORAGE 
10 DEVICES", filed concurrently herewith, which was commonly assigned or subject to an 
obligation of assignment to the same person at the time of invention. 

FIELD OF THE INVENTION 
This invention relates generally to data transfer and storage and more specifically 
IS to a method and system for sharing a single storage resource among a plurality of 
parallel processes. 

BACKGROUND OF THE INVENTION 

20 A computer or a network of computers may be connected to one or more backup 

storage devices to provide greater amounts of low cost storage onto which the 
computers can create archival or backup copies of their files for later recovery if the 
original files are lost or corrupted. Typically, data is copied from a computer first to a 
primary storage device and subsequently copied from the primary storage device to a 

25 lower cost; higher density secondary storage device such as a magnetic tape or an 
optical disk. T3^ically, the secondary storage device is slower than the primary 
storage device. The transfer of data from the primaiy storage device to tiie secondaxy 
storage device typically begins when the amount of used storage space on the primary 
storage device equals or exceeds a predetermined amount or percentage, known as a 

30 "high water mark". 

The purpose of using a "high water mark" is to reduce the chance that substantially 
all of the memory on the primary storage device will be used up, ttiereby causing the 
primary storage device to become inoperative. The high water mark is set to a value 
which is based upon the rate of data coming into the primary storage device and the 

35 rate of data transfer from the primary storage device to the secondary storage device. 
Since the rate of incoming data to the primary storage device usually exceeds tiie rate of 
data transferred from that device to a secondary storage device, tiie high water markr in 
effect; creates a buffer so that the memory of the primary storage device is not used up, 
causing that device to become inoperative. 
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Even with this buffer area, if the rate of data being written to tfie pximary storage 
device exceeds the rate of data being transferred from that device to the secondary 
storage device, the primary storage device will become inoperative. Moreover, setting 
a low "high water mark" may reduce the chance that this will occur, at the expense of 
wasting storage resources on the primary storage device. However, even a low "high 
water mark" can not guarantee that the primary storage device will not fill up. 

Additional problems arise when high water marks are used in a parallel 
processing computer system. Since the number of parallel processes is directly related 
to the rate of incoming data to tfie primary storage device, i.e. the more processes the 
faster the rate, the high water mark should be recalculated and adjusted as the number 
of parallel processes changes. These recalculations can be quite cumbersome and time- 
consuming and could still ultimately be inadequate. 

On the other hand, not recalculating the high water mark when the number of 
processes changes can produce other problems. If the high water mark is not 
recalculated when the nim:jber of processes increases, then the rate of incoming data to 
the primary storage device may exceed the rate of data transfer from the primary 
device to the secondary device, because more processes are writing to the storage 
device. Similarly, when the number of processes decreases, not recalculating the high 
water mark may lead to wasted space, because the high water mark is set too low, 
creating a buffer having a larger size titan necessary. 

SUMMARY OF THE DsA^NTION 
This invention provides a method and apparatus for sharing a single storage 
device among an arbitrarily large number of parallel processes with typically no 
wasted memory space and continued operation even when an error condition or 
interrupt occurs. Briefly, according to the invention, in a networked computer system 
having a communication network cormecting a primary storage device between one or 
more computer devices and a secondary storage device, when some type of interrupt is 
generated or an error condition occurs or is otherwise indicated, a transfer of data from 
primary to secondary storage is initiated. The transfer may be initiated by setting a 
state indicator having at least a non-backup state and a backup state to tfie backup state 
to indicate tiiat a data transfer from the primary storage device to the secondary storage 
device should occur. Alternatively, the transfer may be initialed by begiruiing the 
transfer of data from the primary to the secondary storage device without the use of a 
state indicator. 

For example, when a process attempts to write data to a disk and encounters an 
error condition because there is insufficient disk space available to accommodate the 
write request that process performs one of the following three actions. It begins 
transferring data from the primary storage device to the secondary storage device 
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without setting a state indicator. Alternatively, it begins transferring data from the 
primary storage device to the secondary storage device and sets a state indicator to tfie 
backup state to alert other processes tiiat such a data transfer should occur. Otherwise, 
die process sets &e state indicator to the backup state and does not begin transferring 

S data from the primary storage device to the secondary storage device. 

Parallel processes may be used to transfer data among devices connected 
to the network. Preferably, tiie number of processes working on transfeiring data from 
the primary storage device to the secondary storage device will be less than or equal to 
a predetermined maximum number of processes. The other processes will transfer data 

10 from the computer to the primary storage device. When a state indicator is used, the 
process which completes the transfer of data from the primary storage device to the 
secondary storage device then resets the state indicator to its normal state. Although, 
setting a maximum number of processes for performing data transfer operations 
between the primary storage device and the secondary storage device increases 

IS efficiency, the invention may be used without setting such a limit on the number of 
processes. Preferably, processes which are already executing at the time when the 
interrupt or error condition occurs and did not themselves encounter the error or 
interrupt continue to execute to completion. 

Since a portion of ^e primary storage device's memoiy is not allocated as a 

20 memory buffer, the invention typically eliminates wasted memoiy on the primary 
storage device and eliminates the other problems inherent in the use of a high water 
mark approach in a highly parallel environment Unlike prior systems employing a 
high water mark approach, the invention handles error conditions or interrupts by 
changing to a backup state and performing archiving operations, rather than causing 

25 the system to fail, and without affecting other processes that may exist Thus, the 

invention allows an arbitrary number of parallel processes to execute without the need 
for recomputation and interprocess communications when a process is added or 
removed, thereby permitting a substantially unlimited number of parallel processes to 
share a single storage device. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and further advantages of the invention may be better understood by 
referring to the following description in conjunction with the accompanying drawings, 
35 in which: 



Figure 1 illustrates a general view of the networked computer system; 
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Figure 2 shows a flow chart of the process steps of a metiiod according to the invention 
without the use of a state indicator; 

Figures 3A and 3B show different states of operation of a primaiy storage device; and 

Figure 4 shows a flow chart of the process steps of a method according to the invention 
witii the use of a state indicator. 



DETAILED DESCRIPnON OF ILLUSTRATIVE EMBODIMENT 

Referring to FIG. 1 of the drawings, reference numeral 10 designates generally a 
networked computer system having a communication network 12 linking together at 
least one primaiy storage device 14, at least one secondary storage device 16 and at 
least one computer device 18. Communication network 12 can be a local-area network, 
high-speed bus or other interconnecting mechanism for exchanging messages and data, 
such as AppleTalK Ethernet or Token Ring. 

Storage devices 14 and 16 can each be a specialized storage device designed for 
tiie efficient storage, archival and retrieval of data, or can be a computer augmented 
with greater storage volumes and devices or can be a minicomputer or large computer 
providing storage service in addition to other functions. A primary storage device 14 
has a memory 21 and a central processing unit ("CPU") 22. An external memoiy 
storage unit 23 having a memory 24 may be connected to the primaiy storage device 14. 
Preferably, the secondary storage device 16 is a parallel processor such as a Cray Y- 
MP2E/232 (Cray Research, Cray Research Park, Eagan, MN) connecting with one or 
more external storage devices 20 such as a tape robot 20a or an optical disk unit 20b. 

Computer devices 18 can be any of personal computers, workstations, 
minicomputers or large computers, or other specialized computing devices or 
peripherals attached to the communication network. A computer device 18a includes a 
memory 25 and a central processing unit ("CPU") 26. A computer device 18a may 
include a plurality of disks (not shown) or otfier storage resources, each having a 
memory. 

As shown in TIG. 2, a process or a plurahty of parallel processes are created to 
transfer data among computer devices, primary storage devices and secondary storage 
devices. At block 27, if there is a computer device to be backed up, then at block 28, 
each process attempts to transfer data from a computer device to a primary storage 
device. At decision block 29, if an error condition or interrupt was encountered during 
that data transfer, tiien, at block 30, that process begins transfemng data from the 
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primary storage device to the secondary storage device or tfiat process creates another 

process to perform ^e data transfer. 

At decision block 31, if an error condition or intermpt was encountered during 

the transfer of data from the primary storage device to the secondary storage device, 
5 ttien tiiat process is terminated. Alternatively, tfiat process could try to backup anott\er 

computer device at decision block 27. If at decision block 31 an error condition was not 

encountered, titen at decision block 32 if there is more data to transfer from the primary 

storage device to the secondary storage device, data is transferred at block 30. If at 

decision block 32 there is no more data to transfer, then if at decision block 27 there is 
10 another computer device to be backed up, data is transferred from that computer 

device to the primary storage device at block 28. If at decision block 27 there are no 

more computer devices to be backed up then the process is terminated. If at decision 

block 29 an error condition or interrupt was not encountered then the process tries to 

backup another computer device at block 27. 
15 FIGS. 3A and 3B show a state indicator 38. In FIG. 3A the state indicator 38 is 

stored in the memory 24. Alternatively, as shown in FIG. 3B, the state indicator 38 may 

be stored in memoiy 21. 

The state indicator 38 has at least a first state and a second state, the second state 

is a backup state indicating that data should be transferred from the primary storage 
20 device 14 to the secondary storage device 16. For example, the state indicator may be a 

boolean value having a first state equal to zero and a second state equal to non-zero. 

The state indicator 38 may have any number of states, as long as at least one of the 

states indicates that a data transfer from the primary storage device 14 to the secondary 

storage device 16 (FIG. 1) should occur. 
25 The state indicator 38 is set to the backup state when an interrupt or error 

condition occurs or is ottierwise indicated. 

Preferably, memory 21 also contains a number of processes indicator 40, 

indicating the number of processes which are currentiy performing data transfer 16. 

When flie state indicator 38 is not in the backup states i.e. a transfer from the primary 
30 storage device to the secondary storage device is not indicated, the number of 

processes indicator is preferably set to a value such as zero, a null value, etc. which 

indicates that there are not any processes performing data transfer operations from tfie 

primary storage device to the secondary storage device 

In use as shown in FIG. 4, a state indicator 38 and a number of processes 
35 indicator 40 for indicating the number of processes performing data transfer operations 

from a primary storage device to a secondary storage device are initialized.at box 45. 

CPU 22 starts a process or a plurality of parallel processes at box 46 for backing up tiie 

data in memory 25 of a computer 18a. These processes may be started at any time, box 

47, and typically are created in such a way as to substantially continuously perform 
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backup operations. The number of processes for performing such backup operations 
can be determined by the length of time it takes for each process to complete its 
operations or tasks^ and to a lesser extent by the ratio of data produced by each process 
to the storage capacity of the primary storage device 12. 

At decision block 50, each process checks the value of the state indicator 38. If 
the state indicator 38 is not in the backup state, tiie process at box 51 transfers data from 
tiie computer device 18a to the primary storage device 12. If at decision block 54 an 
error condition occurs or is otherwise indicated or some other type of interrupt occurs, 
the state indicator is set to tiie backup state. Otherwise, the process terminates or, 
alternatively, performs backup operations, box 55, on another computer device, disk, or 
other storage resource. 

On the other hand, if at decision block 50 tiie state indicator 38 is in the backup 
state, then at decision block 52 tiie process compares tfie number of processes indicator 
40 to a predetermined maximum number of processes value. This maximum value 
may depend on the type of external storage devices 20 (FIG. 1) connected to the 
secondaxy storage device 16. If the device 20 is a tape drive, then flie maximum 
number of processes can depend on the number of read/write heads on that drive, 
since each process generally requires two read/ write heads. If the device 20 is an 
optical diskr tiien the number of processes can be limited by the throughput and speed 
of &e device. 

At decision block 52^ if the number of processes indicator 40 is less than the 
maximum number of processes, the process will increment tiie number of processes by 
one, box 53, and then transfier data from the primary storage device 14 to. the secondary 
storage device 16, box 57. At decision block 56, if the data transfer was completed 
without an error condition or interrupt occurring and there is more data to transfer at 
decision block 58, the process continues transferring data from the primary to the 
secondary storage device. If at decision block 58 there is not any more to transfer from 
the primaiy to tiie secondaiy storage device, ttie state indicator 38 and the number of 
processes indicator 40 are resets box 59, and the process terminates. If at decision block 
56 an error condition or interrupt was encountered then the process terminates. 

If at decision block 52 the number of processes indicator 40 is equal to or exceeds 
the maximimn number of processes value, then the process will transfer data from the 
memoxy 25 of the computer device 18 to the memory 21 of the primary storage device 
14. 

There are other equivalent means for determining whether the number of 
processes currently transferring data from storage device 14 to storage device 16 has 
reached a predetermined maximum number of processes. For example, a coimter 
could be set to the maximum number and then decreased each time befox« a new 
process begins transferring data from device 14 to device 16. When the counter reaches 
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zero, the xnaximuin has been reached. There are other equivalent means which would 
be obvious to someone skiUed in the relevant art and which fall within the spirit and 
scope of Ais invention. Additionally, it is possible to implement the invention without 
tracking the number or state of concurrent processes. For example, tfie number of 
* 5 pxocesses may be determined by hardware or otfier resource limitations, rather than by 
a niunber of processes indicator 40. 

If the process performs backup operations which complete the data transfer from 
the primary storage device 14 to the secondaiy storage device 16, the process resets tiie 
state indicator 38 to a state other than the backup state, preferably the first state, and 

10 resets the number of processes indicator 40 to indicate that no processes are currently 
transferring data from device 14 to device 16. Preferably, indicator 40 is set to zero. 

If at decision block 52 the number of processes transferring data from the 
primary to the secondary storage device are equal to or greater than the maximum 
number allowed, tiie process transfers data from the computer device 18a to the 

IS primary storage device 12. If at decision block 54 an error condition occurs or is 

otherwise indicated or some o^er type of interrupt occurs, the state indicator is set to 
the backup state, box 55. 

Preferably, processes which are performing data transfer operations from a 
computer device to the primary storage device 12 continue to perform these operations 

20 even after the backup state is indicated, as long as ttiere is available storage space on 
the primary storage device to accommodate the data transfers. 

The foregoing description has been limited to a specific embodiment of this 
invention. It will be apparent; however, that variations and modifications may be 
made to the invention with the attainment of some or all of its advantages. Therefore, it 

25 is die object of the appended claims to cover all such variations and modifications as 
come within the true spirit and scope of the invention. 
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CLAIMS 

We Oaiin: TT- 

1. A method for transferring data in a networked system having a computer device 
coupled to a first storage device having a memory and connecting to a second storage 
device, said method comprising the steps of: 

transferring data from a computer device to a primaiy storage device until an 
error occurs; and 

transferring data from a primaiy storage device to a secondaiy storage device 
after an error occurs. 

Z A method defined in claim 1 further comprising the steps of: 

setting a state indicator from a first state to a second state when an error occurs 
or is otherwise indicated, the second state indicating that a data transfer from the first 
storage device to the second storage device should occur; and 

checking the value of the state indicator and if tiie state indicator is not in the 
second state, transferring data from a computer device to the first storage device, 

and if fkte state indicator is in the second state, transferring data from tiie first 
storage device to the second storage device. 

3, A method for transferring data in a networked system having a compniter device 
coupled to a first storage device having a memory and coimecting to a second storage 
device, said method comprising tiie steps of: 

setting a state indicator from a first state to a second state when an error occurs 
or is otherwise indicated, tiie second state indicating that a data transfer from the first 
storage device to the second storage device should occur; and 

creating new processes, each process checking the value of tite state indicator 

and if the state indicator is not in tiie second state, transferring data from a 
computer device to tiie first storage device, 

and if the state indicator is in tfie second state, traraferring data from the finst 
storage device to the second storage device. 

4. A method for transferring data in a networked system having a computer device 
coupled to a first storage device having a memory and connecting to a second storage 
device, said method comprising the steps of: 

counting the number of processes currently transferring data from the first 
storage device to tiie second storage device; 
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setting a state indicator from a first state to a second state when an error ocoirs 
or is otherwise indicated/ the second state indicating that a data transfer from the first 
storage device to the second storage device should occur; and 

creating new processes, each process checking tfie value of the state indicator 
5 and if the state indicator is not in the second state, transferring data from a 

computer device to the first storage device, 

and if the state indicator is in the second state, comparing the number of 
processes currently transferring data from the first storage device to the second storage 
device to a predetermined maximum number of processes, 
10 and if that current number of processes transferring data from the first 

storage device to the second storage device is greater than or equal to the maximum 
number of processes, setting ihe state indicator to a state other than the second state 
and transferring data from a computer device to the first storage device, 

and if that current number of processes transferring data from the first 
15 storage device to the second storage device is less than the maximum number of 

processes, transferring data from tiie first storage device to tiie second storage device. 

5. A method defined in claim 4 wherein said error condition occurs during a write 
operation to said memory of said first storage device. 

20 

6. A method for transferring data in a networked system having a computer device 
coupled to a first storage device having a memory and connecting to a second storage 
device, said method comprising the steps o^ 

transferring data from a computer device to a primary storage device until an 
25 interrupt occurs; and 

transferring data from a primary storage device to a secondary storage device 
after an interrupt occurs. 



7. A me&od defined in claim 1 furttier comprising the steps of: 
30 setting a state indicator from a first state to a second state when an interrupt 

occurs or is otherwise indicated, the second state indicating tiiat a data transfer from 

itie first storage device to the second storage device should occur; and 

checking the value of ttie state indicator and if the state indicator is not in the 

second state, transferring data from a computer device to the first storage device, 
35 and if the state indicator is in the second state, transferring data from the first 

storage device to the second storage device. 
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8. A method for transferring data in a networked S3^tein having a computer device 
coupled to a first storage device having a memoiy and connecting to a second storage 
device^ said method comprising the steps of: 

setting a state indicator from a first state to a second state when an interrupt 
occurs or is otherwise indicated, the second state indicating that a data transfer from 
the first storage device to the second storage device should occur; and 

creating new processes, each process checking the value of tfie state indicator 

and if the state indicator is not in the second state, transferring data from a 
computer device to the firststorage device, 

and if the state indicator is in the second state, transferring data from the first 
storage device to &e second storage device. 

9. A method for transferring data in a networked system having a computer device 
coupled to a first storage device having a memoiy and connecting to a second storage 
device, said method comprising the steps of: 

counting the number of processes currently transferring data from the first 
storage device to the second storage device; 

setting a state indicator from a first state to a second state when an interrupt 
occurs, the second state indicating tiiat a data transfer from the first storage device to 
the second storage device should occut; and 

creating new processes, each process first checking &e value of the state 
indicator 

and if the state indicator is not in the second state, transferring data from a 
computer device to the first storage device 

and if the state indicator is in the second state, comparing the ntunber of 
processes currently transferring data from the first storage device to the second storage 
device to a predetermined maximum number of processes, 

and if that current number of processes transferring data from the fust 
storage device to the second storage device is greater than or equal to the maximum 
number of processes, setting the state indicator to a state ottier ttian the second state 
and transferring data from a computer device to the first storage device;, 

and if tiiat current number of processes transfeiring data from the first 
storage device to tiie second storage device is less than the maximum number of 
processes, transferring data from the first storage device to the second storage device. 

10. An apparatus for transferring data in a networked computer system, said 
apparatus comprising: 

a first storage device having a central processing unit and a memory; 
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a second storage device having a plurality of external memory devices connected 
ttiereto; 

a plurality of computer devices having a central processing unit and a memory; 

means for networking together said first storage device, said second storage 
5 device and said plurality of computer devices, said networking means permitting the 
transfer of data from said plurality of computer devices to said first storage device and 
from said first storage device to said second storage device; 

a state indicator in said memoiy of said first storage device, said state indicator 
having at least a first state and a second state, said second state indicating the need for 
10 a data transfer from said first storage device to said second storage device; and 

means for setting said state indicator to said second state when an error occurs 
or is otherwise indicated. 



11. An apparatus for transferring data in a networked computer system, said 
15 apparatus comprising: 

a first storage device having a central processing unit and a memory; 

a second storage device having a plurality of external memory devices connected 
thereto; 

a plurality of computer devices having a central processing urdt and a memory; 
20 means for networking together said first storage device, said second storage 

device and said plurality of computer devices, said networking means permitting the 
transfer of data from said plurality of computer devices to said first storage device and 
from said first storage device to said second storage device; 

a state indicator in said memory of said first storage device, said state indicator 
25 having at least a first state and a second state, said second state indicating the need for 
a data transfer from said first storage device to said second storage device; 

means for setting said state indicator to said second state when an error 
condition is encoimtered during a write operation to said memoiy of said first storage 
device; and 

30 means for counting tiie number of processes currently traitsferring data from 

said first storage device to said second storage device and comparing that number to a 
predetermined maximum number. 
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